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ABSTRACT 

This  paper  describes  the  results  of  a research  done  in  the  Royal  Norwegian  Navy  into  the  decision  structure  of  the 
evaluation  process  of  investing  in  technology-based  training  systems.  A comprehensive  methodology  was  developed, 
that  evaluates  technology-based  training  systems  as  information  systems.  The  evaluation  method  can  reveal  cost  driving 
training  objectives  and  make  sure  focus  on  the  strategic  value  of  the  information  system  components  is  maintained. 

INTRODUCTION 

The  Royal  Norwegian  Navy  (RNoN)  is  currently  evaluating  investments  in  new  operational  equipment.  Investments  in 
technology-based  training  systems  are  a part  of  these  investments.  The  RNoN  has  developed  a cost/benefit  analysis 
method  that  is  particularly  suited  to  evaluate  technology-based  training  systems.  The  investment  evaluations  conducted 
so  far  have  merely  focused  on  the  training  needs. 

A more  comprehensive  methodology  has  now  been  developed,  that  evaluates  technology-based  training  systems  as 
information  systems,  and  also  takes  account  of  their  technological  and  life-cycle  consequences.  The  evaluation  method 
was  developed  by  extensive  literature  investigation  and  case  studies.  The  outcome  of  the  case  studies  was  used  to 
improve  the  decision  structure  of  the  evaluation  method.  This  method  was  developed  in  a research  co-operation  between 
Delft  University  of  Technology  in  The  Netherlands  and  the  Naval  Trainmg  Establishment  HNoMS  Tordenskjold  in 
Norway. 

TECHNOLOGY-BASED  TRAINING  SYSTEMS 

This  term  can  best  be  defined  by  other  definitions.  An  information  system  can  be  defined  as  a collection  of  software, 
hardware,  people,  procedures,  and  data  (Brussaard,  1994).  Further,  an  instructional  system  can  be  defined  as  an 
arrangement  of  resources  and  procedures  used  to  promote  training  (Gagne,  Briggs,  & Wagner,  1992).  In  this  research  the 
context  of  information  systems  was  within  the  instructional  system  where  information  systems  are  used  to  promote 
training.  Training  is  the  learning  or  acquisition  of  skills,  knowledge  and  attitudes  in  order  to  enhance  performance  at  a 
given  task  (Bumiston  & Tayler,  1995). 

In  this  paper  the  term  technology-based  training  implies  the  use  of  information  systems  in  any  part  of  the  instructional 
system.  To  emphasise  that  an  information  system  is  used  to  promote  trainmg,  the  term  chosen  is  technology-based 
training  system.  This  implies  that  technology-based  training  systems  must  also  consist  of  the  five  information  system 
components. 


Paper  presented  at  the  RTO  HFM  Workshop  on  "The  Capability  of  Virtual  Reality  to  Meet  Military 
Requirements ”,  held  in  Orlando,  USA,  5-9  December  1997,  and  published  in  RTO  MP-54. 
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Case  1:  Communication  Trainer 

The  communication  trainer  was  used  in  the  training  of  internal  and  external  operational  communication  procedures.  The 
communication  trainer  is  an  integrated  part  of  the  existing  technology-based  training  systems  at  the  Naval  Training 
Establishment  HNoMS  Tordenskjold.  The  main  objectives  of  the  communication  trainer  are:  to  take  care  of  all  types  of 
training,  covering  procedures  and  plans  for  using  existing  operational  communication  equipment.  Secondly,  internal 
communication  between  trainees  and  trainees/instructor.  The  third  objective  was  to  provide  a standard  platform  for  voice 
communication  for  existing  and  future  technology-based  training  systems  within  the  RNoN. 

Case  2:  Scenario  Management  System 

The  scenario  management  system  aims  to  improve  the  instructor’s  working  environment  by  providing  the  instructor  with 
the  required  tools  for  supporting  the  training  process.  The  system’s  main  objectives  were  to  enable  re-design  of  the 
instructor  function  to  reduce  the  number  of  instructors  involved  in  the  training  process.  Further,  it  was  intended  to 
develop  a standard  instructor  workstation  and  a prototype  of  an  interconnection  device  for  the  Navy’s  future  technology- 
based  training  systems. 

THE  EVALUATION  METHOD 

The  idea  behind  the  evaluation  method1  described  in  this  paper  is  to  provide  a structured  approach  to  the  decision  process 
of  investing  in  technology-based  training  systems.  An  analysis  of  training  needs,  information  system  components  and 
costs  are  required  before  a decision  can  be  made  on  what  investment  strategy  to  pursue.  These  analyses  are  important  to 
prevent  overlooking  unfavourable  consequences  in  the  initial  enthusiastic  planning  phase. 

In  the  context  of  evaluation  of  technology-based  training  systems  the  term  value  is  defined  as  the  sum  of  associated  costs, 
benefits  and  risks  of  the  desired  competence  level  and  its  support  technology.  In  other  words,  value  can  be  structured  into 
the  three  components  of  (1)  training  domain  evaluation,  (2)  technology  domain  evaluation  and  (3)  cost  evaluation.  These 
evaluations  can  be  done  by  means  of  the  method  outlined  in  Figure  1.  Consequently,  to  determine  the  value  of  different 
investment  strategies  requires  a structured  approach  by  the  RNoN  organisation.  The  different  elements  of  the  method 
(Figure  1)  are  described  in  the  following  sections. 

Identify  System  & Search  Space 

When  a particular  problem  is  perceived,  it  is  of  course  important  to  be  sure  that  an  associated  training  need  exists.  It  may 
be  that  an  apparent  need  can  be  satisfied  more  economically  in  some  other  way,  for  example  by  a change  in  the 
organisation  or  the  introduction  of  a different  personnel  policy  (Akerjordet,  1993;  Patrick,  1 992). 


X , 

Comparison  of  Alternatives  | 


Figure  1.  Method  for  investment  evaluation  of  technology-based  training  systems  (Svendsen,  1996) 


1 The  term  often  used  for  the  evaluation  process  of  technology-based  training  systems  is:  training  needs  analysis  (TNA)  (I1MS0  247,  1992;  RNoN 
NSST,  1995;  Gramson,  1995). 
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Identification  of  needs,  problems,  and  opportunities  are  done  in  order  to  trace  valuable  technology-based  training  system 
solutions  for  the  organisation  (van  Reeken,  1994).  The  identification-process  can  be  carried  out  on  the  basis  of  the 
information  system  strategy  formulation  principle  described  by  Earl  (1989).  It  can  be  done  in  three  complementary  ways: 
top-down,  bottom-up,  and  inside-out.  This  process  can  also  be  used  to  evaluate  technology-based  training  system 
investment  strategies.  The  way  this  identification  process  is  performed  is  by  means  of  an  approach  strategy.  The  three 
main  approach  strategies  are;  (1)  clarification  of  organisational  goals,  (2)  evaluation  of  existing  investments  and  (3) 
innovation  by  application  of  new  technology.  A fourth  approach  strategy  is  to  combine  the  three  strategies  in  various 
ways.  The  strategies  of  differentiation  and  focus  (Porter,  1985)  can  be  used  to  further  improve  the  scope  of  the  evaluation 
approach. 

When  a discrepancy  in  the  form  of  a need,  problem,  or  opportunity  is  identified,  further  analysis  can  be  undertaken.  For 
this  analysis,  a mission  statement  or  a project  goal  must  be  defined.  A mission  statement  provides  a long-term  continuity, 
because  it  gives  the  organisation  and  its  employees  direction  and  purpose  (Ishoy  & Swan,  1992). 

Task  Analysis 

Task  analysis  is  an  analysis  of  the  final  task  for  which  training  is  to  occur.  In  other  words,  it  derives  the  main  elements  of 
the  training  content  (Patrick,  1992).  Without  the  task  analysis,  there  can  be  no  guarantee  that  the  training  given  will  not 
be  deficient  in  some  way  (ENTWG/TT-PUB  2,  1983).  In  addition  to  the  analysis  of  the  tasks  to  be  trained,  the  task 
analysis  must  also  include  the  instructor  tasks.  From  the  Cases  the  task  space  also  includes  the  instructor  tasks.  By 
including  the  instructor  tasks  in  the  task  space  of  the  planned  technology-based  training  system,  a more  complete  picture 
of  the  system  can  be  created. 

A job  is  the  duties,  tasks  and  sub-tasks  performed  by  one  individual  (ENTWG/TT-PUB  1,  1990).  Where  the  term  task  is 
any  set  of  activities  occurring  at  about  the  same  time  sharing  the  same  common  purpose  that  is  recognisable  by  the  task 
performer  (Patrick,  1992). 
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Figure  2.  The  task  hierarchy  of  the  scenario  management  system 


The  task  can  be  broken  down  into  sub-tasks,  which  are  parts  of  a task  that  may  not  stand  alone.  When  dealing  with  task 
analysis  it  is  important  to  know  what  is  meant  by  it,  and  how  it  is  achieved  and  carried  out. 

The  task  analysis  technique  used  in  the  evaluation  method  was  based  on  an  object-oriented  approach  (Bots,  1989).  This 
technique  is  a combination  of  the  critical  incident  technique  (Flangan,  1954),  hierarchical  task  analysis  and  Petri  nets 
(Kirwan  & Einsworth,  1992).  The  advantage  of  this  object-oriented  technique  is  its  descriptive  as  well  as  its  prescriptive 
way  of  illustrating  tasks  and  sub-tasks  (Bots,  1989). 

Figures  2 and  3 give  a high  level  description  of  the  tasks  of  the  scenario  management  system  of  Case  2.  Figure  2 
illustrates  the  task  space.  Figure  3 illustrates  the  relationship  between  the  different  sub-tasks.  This  object-oriented  task 
analysis  technique  may  help  to  reduce  differences  in  terminology  and  create  a common  ground  for  understanding 
evaluating  technology-based  training  systems. 

Audience  Analysis 

Analysis  of  the  trainees  will  vary  from  an  analysis  of  main  features  and  down  to  a detailed  analysis  of  the  individual 
trainee  (UD  3-2,  1984).  The  nature  of  the  trainees  must  be  determined  and  their  skills  assessed  (ENTW'G/TT-PUB  2, 
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1983).  The  first  decision  to  be  made,  is  how  much  information  is  necessary  for  an  adequate  audience  identification  or 
analysis.  An  analogy  to  conduct  an  audience  analysis  is  found  within  product  marketing  (Moller  & McDermott,  1992). 
Following  this  idea,  the  trainees  or  target  audience  can  be  divided  into  different  clusters  (Kotler,  1994).  Clustering  of  the 
target  audience  can  be  done  after  different  criteria. 

In  Case  1 the  target  audience  was  clustered  after  its  respective  organisations.  Some  of  these  organisations  were  Navy  and 
Coast  Guard  vessels,  and  coastal  fortresses.  These  organisations  are  situated  at  different  locations  which  implies  a target 
audience  scattered 
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Figure  3.  The  scenario  management  task. 

over  a geographical  area.  Considerations  about  the  access  to  different  clusters  of  the  target  audience  must  be  done  when 
evaluating  technology-based  training  systems. 

Further,  the  target  audience  can  also  be  clustered  after  their  respective  training  courses  and  career  programmes.  When  the 
clustering  of  the  target  audience  is  done  the  assessment  of  the  trainees  characteristics  can  be  performed. 

The  target  audience  analysis  will  account  for  the  users  of  the  system  for  which  training  is  to  take  place.  It  is  necessary  to 
know  at  what  stage  the  trainees  enter  the  training  process  (i.e.  what  skills,  knowledge  and  attitudes  they  possess  prior  to 
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training  Burmston  & Tayler,  1995).  This  is  important  because  a technology-based  training  system  can  be  used  for  both 
initial  training  and  continuation  training  (NIAG(94)SG/45-R4,  1994).  In  the  Royal  Norwegian  Navy  technology-based 
training  systems  were  applied  on  three  main  levels  of  training.  These  were  the  training  of  (1)  individual,  (2)  team  and  (3) 
command  operations  (RNoN  NSST,  1995). 

Competence  Contribution 

Competence  is  the  skills,  knowledge  and  attitudes  required  to  perform  an  operational  task.  Different  technology-based 
training  systems  may  differ  in  meeting  operational  competence  requirements.  To  evaluate  this  competence  contribution, 
training  objectives  have  to  be  defined.  A training  objective  can  be  seen  as  an  expression  that  emphasizes  what  the 
technology-based  training  system  must  be  able  to  support  for  training  to  occur.  Subsequently,  these  training  objectives 
are  statements  of  new  competencies  the  trainee  should  possess  at  the  completion  of  training  to  meet  operational 
competence  requirements  (Russel,  Molenda,  & Heinich,  1993). 

To  determine  the  training  objectives  structured  guidelines  are  available.  One  way  of  detailing  training  objectives  is  by 
means  of  the  convention  called  the  five-component  objective  (Gagne  et  al  1992).  The  five-component  training  objective 
specifies  elements  such  as  the  situation,  required  competence  level,  specific  actions,  tools  and  constraints  (Gagne,  1992). 
Well-defined  training  objectives  are  essential  when  determining  the  training  method  to  be  used. 

The  training  objectives  can  also  be  used  to  gain  information  on  the  training  system’s  temporal  aspects.  In  Case  1 the 
following  questions  were  asked.  In  what  time  perspective  are  the  training  of  these  tasks  valid?  Further,  will  changes  in 
the  operational  organisations  in  the  near  future  have  any  impact  on  the  training  objectives?  Temporal  analysis  is 
important  for  the  future  effectiveness  of  the  technology-based  training  system. 

In  addition  to  the  training  objectives  in  Case  1,  instructor  objectives  were  defined.  The  instructor  objectives  were  also  a 
natural  element  in  Case  2,  to  pinpoint  the  instructor’s  needs  in  the  training  process.  The  training  and  instructor  objectives 
are  used  to  evaluate  different  technology-based  training  system  investment  strategies.  The  different  strategies  may  range 
from  simple  to  complex  systems  solutions. 

Training  Commitment 

Organisational  readiness  analysis  is  performed  in  the  Training  Commitment  stage  of  the  evaluation  method.  It  examines 
the  commitment  by  the  subject  matter  responsible  organisation  to  implement  the  proposed  technology-based  training 
system  into  its  training  programmes. 

In  Case  1 an  important  issue  was  improving  the  utilisation  of  existing  technology-based  training  systems  within  the 
RNoN.  An  aspect  of  this  improved  utilisation  is  how  the  planned  technology-based  training  system  fits  into  the  existing 
and  future  training  courses  and  programmes.  Decisions  on  including  the  system  in  the  different  training  programmes 
were  up  to  the  subject  matter  responsible  organisation  and  not  the  service  provider  department.  For  the  subject  matter 
responsible  organisation  it  can  be  a question  of  convenience  and  practicality  before  the  issue  of  training  effectiveness. 
Training  effectiveness  is  the  best  utilisation  of  resources  available  in  bringing  the  trainees  or  target  audience  up  to  the 
required  standard  (ENTWG/TT-PUB  1,  1990). 

Applications 

This  stage  covers  the  transformation  of  revealed  training  and  instructor  objectives  into  an  environment  for  application 
delivery  solutions  in  the  form  of  technology-based  training  systems. 

The  man-machine  interface  (MMI)  is  the  way  the  trainees  and  instructors  communicate  with  the  technology-based 
training  system.  Therefore  it  is  important  that  the  MMI  provides  the  necessary  cues  for  training  to  occur  (Bumiston  & 
Taylor,  1995).  In  the  two  Cases  the  MMI  were  windows-based.  In  Case  1 the  MMI  emulated  the  functionality  and 
presented  the  image  of  the  operational  communication  equipment.  The  purpose-built  hardware  implemented  were  the 
different  types  of  telephones  resembling  the  operational  parent  equipment.  The  evaluation  of  man-machine  interface 
becomes  more  difficult  when  considering  more  complex  technology-based  training  system  solutions.  Here  in  addition  the 
MMI  can  be  purpose-built  hardware  to  replicate  the  operational  equipment,  as  in  Case  1.  The  reproduction  of 
kinaesthetic  cues,  visual  images  and  audio  must  also  be  included. 
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A technology-based  training  system  may  consist  of  different  software  modules.  In  both  Cases  different  system  modules 
were  defined  such  as  database  management  systems  (DBMS),  scenario  generation  and  simulated  entities.  Other  system 
modules  used  in  a technology-based  training  system  can  be  geographical  information  systems  (GIS),  and  modelling  and 
animation  tools.  In  addition  to  these  software  modules  the  operating  system  software  must  also  be  evaluated.  In  the  two 
Cases  both  client  and  server  operating  system  software  were  evaluated. 

Data  Needs  & Requirements 

Data  forms  the  basis  of  every  information  system.  The  most  important  issue  here  is  to  determine  the  data  structure  and 
the  most  important  data-elements  (Eilers,  Dijkstra  & Hoven,  1991). 

In  Case  2 a database  design  was  developed  to  reveal  the  data  foundations  required  by  the  system.  This  database 
incorporated  elements  such  as  scenario  descriptions,  configuration  and  model  parameters,  charts  organisation  and  trainee 
information.  Other  data  elements  considered  were  charts  and  terrain  information  available  on  standard  data  formats. 
Respectively  formats  such  as  DX-90,  digital  terrain  elevation  data  (DTED)  and  digital  feature  analysis  data  (DFAD) 
(RNoNNSST,  1995). 

In  Case  1 and  Case  2 evaluation  of  data  elements  that  must  be  exchanged  with  other  existing  or  planned  technology- 
based  training  systems  were  also  undertaken. 

Computing  & Communication 

It  is  necessary  to  perform  analysis  of  the  required  computing  architecture  of  the  technology-based  training  system.  This 
provides  an  insight  into  the  ability  of  a particular  technology-based  training  system’s  architecture  to  meet  performance 
requirements.  The  computing  architecture  of  technology-based  training  systems  may  range  from  a collection  of  elements 
such  as  image  generators,  audio  systems,  kinaesthetic  control  systems,  specific-purpose  built  components  and  scenario 
management  work  stations,  to  personal  computers  in  a network  or  stand-alone  solutions. 

In  the  Cases,  both  the  internal  communication  network  and  the  external  integration  with  other  technology-based  training 
systems  were  evaluated.  In  Case  2 the  system  operated  in  a distributed  environment  with  a local  area  network  as  central 
communication  media.  In  addition  it  was  supposed  to  exchange  information  with  other  systems  by  means  of  a distributed 
interactive  simulation  (DlS)-gateway. 

People  & Procedures 

To  implement  the  technology-based  training  system  a set  of  procedures  are  required.  These  procedures  or  management 
activities  differ  with  the  life-cycle  state  of  the  system  (Looijen,  1991).  In  both  Cases  management  procedures  were 
outlined. 

In  Case  1 the  management  of  the  technology-based  training  system  was  to  be  carried  out  with  the  existing  staffing  level. 
Human  recourse  evaluations  are  required  for  the  fulfilment  of  these  management  procedures  which  are  necessary  to  keep 
the  training  system  operational.  The  method  supports  the  evaluation  of  required  management  procedures  and  staffing 
considerations  for  technology-based  training  systems. 

Cost  Analysis 

To  gain  insight  into  the  financial  aspects  of  technology-based  training  system  cost  analysis  must  be  undertaken.  From  the 
Cases  the  initial  investment  cost  was  identified.  Further  in  Case  1 the  life-cycle  cost  was  estimated  for  the  different 
investment  strategies.  The  initial  costs  can  be  divided  into  two  categories  (Lien  & Bjorn,  1995).  First  all  the  costs  related 
to  the  planning  and  accomplishment  of  the  project  and  then  the  second  category  covering  all  the  costs  concerning  the 
procurement  of  software,  data,  hardware  and  facilities.  The  life-cycle  costs  regards  all  costs  associated  with  the  training 
system’s  maintenance  and  management  activities. 


CONCLUSION 
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The  effectiveness  of  a technology-based  training  system  is  whether  or  not  it  meets  well  defined  objectives  that  are  based 
on  the  needs  and  constraints  of  its  ownership  organisation.  The  method  suggested  in  this  paper  can  be  a helpful  means  to 
evaluate  this  effectiveness. 

By  means  of  the  evaluation  method  different  investment  strategies  for  technology-based  training  systems  can  be  outlined. 
These  different  investment  strategies  can  be  illustrated  by  means  of  an  evaluation  matrix  (Berghout,  1994).  The  vertical 
and  horizontal  axes  portray,  respectively,  the  training  and  technology  value  of  the  investment  strategy.  The  area  of  the 
circle  reflects  the  initial  and  life-cycle  cost. 


Figure  4.  The  investment  strategy  evaluation  matrix  (Berghout,  1994). 

The  training  value  of  a technology-based  training  system  is  its  effectiveness,  efficiency  and  relevance  to  the  operational 
task,  as  perceived  by  its  users.  The  technology  value  is  the  strategic  alignment  of  the  information  system  components 
with  organisational  information  plans. 

Reasons  for  a more  formal  approach  to  investment  evaluations  of  technology-based  training  systems  are: 

• The  large  sums  of  capital  that  are  often  involved  in  procurement  of  technology-based  training  systems. 

• Technology-based  training  systems  often  have  significant  consequences  for  other  technology  investments,  hence 
compete  with  other  investment  opportunities  (Hares  & Royle,  1994;  Berghout,  1994). 

• Formal  approaches  create  a common  ground  for  communication  between  the  different  parties  involved  in  the 
decision  process. 

• Training  needs  and  their  technical  solutions  need  to  be  co-ordinated. 

• Increased  awareness  exists  on  training  objectives  and  its  relevance  to  the  operational  situation. 

• The  initial  enthusiasm  regarding  new  products  tempt  decision  makers  to  overlook  possible  negative  consequences. 

The  investment  analysis  technique  that  has  been  presented  in  this  paper  seems  to  provide  a comprehensive  evaluation  of 
technology-based  training  systems. 
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